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Abstract 

The purpose of this study was to identify risk factors 
associated with unsuccessful discharge status of students with 
serious emotional/behavioral disorders (E/BD) from a secondary 
level, therapeutic day treatment program. In addition, this 
article describes and demonstrates the effective use of 
epidemiologic analysis as a program evaluation method by comparing 
risks associated with identified characteristics of the E/BD 
population to program completion status. 

Results indicated significant differences in successful 
program completion related to individual variables, specifically 
student substance abuse and involvement with Juvenile Services. 
Further, we found that additive effects of several variable pairs 
greatly increased the risk of unsuccessful program outcomes. The 
variable pairs identified to relate to the highest risk of 
unsuccessful outcomes (between three and seven times more likely) 
included (a) female and Caucasian, (b) female and truant, (c) 
female and substance abuse, (d) female and Juvenile Services 
involvement, (e) Black and male, (f) Caucasian and substance abuse, 
(g) Caucasian and Juvenile Services involvement, and (h) truant and 
substance abuse. 

Implications for E/BD program development and ongoing program 
evaluation with epidemiologic analysis were addressed. 
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Application of Epidemiology to an Outcome Based Evaluation 
of an Educational Program Serving Emotionally Disturbed Students 

The prevalence of behaviorally/emotionally disturbed (B/ED) 
school-aged children identified as needing services in the United 
States was estimated to be 3 to 6 percent in 1989 (National 
Mental Health Association [NMHA] , 1993) • The education of youth 
with serious emotional and behavioral disorders (E/BD) is 
considered one of the most difficult challenges facing our 
schools today • Much of our existing programming is inadequate, 
with students with E/BD having the least positive outcomes of any 
group of children with disabilities (Lichtenstein, 1988; NMHA, 
1993; Sitlington, Frank, & Carson, 1990) • The school success of 
students with E/BD is unsatisfactory as these students are 
reported to have the highest dropout rate of any group of 
children with disabilities (Lichtenstein, 1988) • National 
surveys indicate that the dropout rate for students with E/BD is 
currently 50%, with only 39% actually earning diplomas • 
Moreover, the students with E/BD who remain in school are found 
to have lower grades and fail more often than any other students 
(NMHA, 1993) . These students are also more likely, than any 
other students with disabilities to be placed in restrictive 
settings (NMHA, 1993; United States Department of Education, 
1993) . 

In the quest for information relevant to serving at-risk 
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AERA: EPIDEMIOLOGY EVALUATION 3 
populations, educational researchers have pursued 
multidimensional models using multivariate statistics which have 
yielded few concrete results. The answers provided in the 
literature are generally neither clear nor easily understood. 
Instead, educational programs serving target populations of 
challenged students have a need to know which variables identify 
students as being at-risk for successful or unsuccessful outcome 
from their programs. With this information, program developers 
and administrators would be able to identify and intervene early 
in an at-risk student's enrollment to better the student's 
chances of success. Further, such riata serve to provide 
descriptive formative and summative evaluation information 
regarding a program's past or current performance standards 
relative to the population being served. The problem with such 
data, however, resides in the methods of analysis and 
dissemination to practioners. 

The field of epidemiology has recently been edging its way 
into the social sciences. It has become a useful tool in basic 
applied developmental research (Carran & Scott, 1992) by 
presenting simple answers to practitioner's questions. 

The purpose of this paper is to present the results of a 
developmental epidemiological analysis to quantify risk factors 
associated with unsuccesL'f ul completion of a day treatment 
program serving students with E/BD in a segregated facility. 
This analysis addresses the process of identifying, analyzing, 
and interpreting the risk factors associated with poor outcome. 




«) 



AERA: EPIDEMIOLOGY EVALUATION 4 
for use as a program evaluation tool. 
Epidemiology 

Epidemiology is defined as "The study of the distribution 
and determinants of health-related states and events in 
populations, and the application of this study to control of 
health problems." (Last, 1983, p. 33). This is a very broad 
definition for a methodology which has been used as a tool for 
description, intervention, and evaluation of biological states 
around the world. Since Sir John Snow (1813 - 1858) wrote the 
first definitive text on epidemiology and made an explicit 
statement of germ theory transmission (30 years before it was 
discovered by Koch), there have been many definitions. In the 
past 50 years the definition has broadened from concern with 
communicable disease epidemics to take in all phenomena related 
to the health of a population. 

There are four fields of study in Epidemiology: (a) 
analytic; (b) descriptive; (c) experimental; and (d) 
developmental. Each will be defined and discussed. 

Analytic epidemiology is a type of hypothesis testing study 
to investigate causative factors related to a disease or 
dependent variable. An example of analytic epidemiology which 
has recently been in the media is the study of the long term 
effect of exposure to the herbicide Agent Orange. The hypothesis 
being tested was the relationship between the Agent Orange toxin 
and the prevalence of cancer. Using a Case-Control design, 
investigators identified Viet Namn Armed Forces service personnel 
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AERA: EPIDEMIOLOGY EVALUATION 5 
exposed to the Agent Orange toxin (cases) and service personnel 
serving in Viet Namn at the same time who had not been exposed to 
the toxin (controls) . These individuals were tracked to 
determine their current health status. The number of cases with 
cancer were compared to the number of controls with cancer, and 
calculations were performed to determine the relationship between 
the Agent Orange toxin and the prevalence of cancer. 

Descriptive epidemiology is probably the most easily 
recognizable form of epidemiology, and the form that makes the 
papers most often. This field of epidemiology describes the 
occurrence of disease in a population: person, place, and time. 
Descriptive epidemiology determines what caused some of the 
people at the picnic to get sick; the trail back to the potato 
salad or the shrimp salad. It was the technique used to track 
down the contaminated and undercooked hamburger which killed a 
number of people last summer in the upper northwest United 
States. It is the tool of choice used by Public Health Officials 
since it is specifically targeted to an outcome, uses personal 
interview techniques, and generally arrives at a remediable 
conclusion. 

The third type of epidemiology is Experimental epidemiology. 
As its name implies, this is a rigidly controlled and detailed 
experimental study procedure which generally takes place in 
randomized controlled trials. Drug trials are the best 
exemplars. When the National Institute of Health (NIH) tests a 
new cancer drug, a sample population is identified and qualified. 
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The drug is administered to an experimental group whose 
biological results are then compared to a control group. 

The fourth type of epidemiology is Developmental 
epidemiology, which investigates the distribution of behavioral 
outcomes in infancy and childhood and the indicators of their 
occurrence. Scott, Shaw, and Urbano (in press) present a 
succinct and effective description of developmental epidemiology, 
linking birth certificate data to later educational disabilities. 
Numerical values were calculated for the relationship between 
birth characteristics (i.e., birth weight, socio economic status, 
etc..) and the prevalence of educational disabilities and called 
risk factors. The numerical value of risk factors represents 
the influence of the risk factor variable on the outcome. A 
discussion of the concept and implications of risk are necessary 
to explain the relevance of developmental epidemiology to program 
evaluation. 

Epidemiological Conceptual Terms. 

Risk Analysis . Risk is defined as the probability that an 
event will occur (Last, 1983). To calculate risk estimates, the 
researcher must first select the dichotomous event to 
investigate. In this study, the outcome is program completion 
and is categorized as poor vs. good. The next step involves a 
definition of the means by which the probability will be 
measured, typically as one or more identified variables which are 
described as present or absent. The presence or absence of each 
variable is then compared to the occurrence of the event, 



AERA: EPIDEMIOLOGY EVALUATION 7 
indicating the likelihood of the event occurring relative to the 
targeted variables. 

As Table 1 shows, developmental epidemiology is a method of 
2x2 categorical data analysis which enables outcome measures to 
be calculated in terms of indices of risk. A risk factor, then, 
is defined as an identified variable which is associated with an 
increased probability of a specified poor outcome, in this case 
the unsuccessful completion of a program. This does not imply a 
causal relationship. Rather, the variable is considered to be an 
indicator of this increased risk for poor outcome. (For a more 
detailed discussion of risk see Carran & Scott, 1992) . 

Once a risk factor is identified, the following information 
may be determined through simple calculations; (a) What 
proportion of subjects in the total sample were identified as 
having the predictor variable present? and (b) Of the subjects 
determined to have experienced a poor outcome, what proportion 
were identified as having the predictor variable present? The 
answers to these two questions are easily answered by stratifying 
the data according to Table 1 and then calculating the Exposure 
Frequency (EF) and Exposure Odds Ratio (OR) , respectively (see 
Table 1 for calculation formulas) . 

Epidemiologic analysis is not a complicated model that 
relies upon theoretical relationships estimated through 
complicated analyses. Rather, it is easy to apply to any 
situation, and analysis/ interpretation of the data are 
straightforward. Like any good evaluation tool, it forces the 
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evaluator to select specific target outcomes, quantify the 
outcomes, and then compare the occurrence of the outcome between 
levels of a predictor variable. It is critical to point out, 
however, that the results of this type of analysis are dependent 
upon the reliability and validity of the outcome variable. 

Additive Model of Interaction. The assessment of risk in 
epidemiology is based on the sufficient-component causal model. 
In this model there are sufficient causes for an outcome to 
occur; some causes are known, others remain to be disco\'ered and 
are not considered random in occurrence. These sufficient causes 
are typically a combination of more than one agent. Each 
sufficient cause is considered to be independent in the sense 
that it can act alone, or it may interact with other agents. 
Thus, a developmental disability could be due to a single 
sufficient cause (i.e. LBW) or due to the single sufficient cause 
(poverty) , or due to the interaction of these two factors. this 
is the Additive Model of Interaction. This will almost always be 
the case for developmental outcome; combinations of agents, 
exposures, and personal characteristics that occur in life 
experience. 

Epidemiologic Evaluation Cycle . Epidemiologic analysis can 
be used as a cyclical evaluation tool by implementing an 
analysis-intervention-analysis assessment schedule. Researchers, 
program developers, and teachers alike can calculate risk 
estimates for a wide variety of situations, design interventions 
intended to reduce risk for individuals with specific 
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characteristics, and then reassess the risk levels at a later 
point in time. This evaluation tool will be discussed in depth 
using results from this paper. 

Selected Risk Characteristics of SEP Adolescents 

The first step in determining factors related to E/FD 
program outcomes was the identification of characteristics most 
likely to be risk factors. Based on a review of the literature 
on characteristics of adolescents with E/BD, we selected five 
risk variables for, investigation of their impact upon outcome: 
juvenile services involvement, truancy, substance abuse, gender, 
and ethnicity. The relation of each to adolescents with E/BD is 
briefly summarized. 

Truancy . Truancy has a significant negative impact on 
students since it interferes with the delivery of educational 
services and may be a precursor to the development of later 
psychological difficulties (Schultz, 1987). In addition, 
attendance in school has been shown to be directly related to 
program completion and likelihood of dropping out in students 
with disabilities (United States Department of Education, 1992; 
1993) . Truants are more likely to engage in delinquent 
activities and deviant behaviors while not attending school, run 
away from home, and score below other non-attenders on measures 
of school achievement (Galloway, 1983; Robbins & Ratcliff, 1980). 
In addition, Zieman and Benson (1980; 1981) found a gender 
difference in the reasons and concerns for truancy in 
adolescents, suggesting the possibility of differential rates and 
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outcomes for males and females. Since attendance has been 
identified as a factor in successful E/BD program completion of 
students in a psychoeducational treatment facility (Leone, 
Fitzmartin, Stetson, & Foster, 1986) , truancy may impede 
successful program completion. 

Juvenile Services Involvement . Involvement by children and 
adolescents in the Juvenile Services Administration system 
indicates delinquent or criminal behavior on the part of the 
child. A common diagnostic category among behavioral ly 
disordered adolescents, especially juvenile offenders, is conduct 
disorder (Kutcher et al., 1989; Wierson, Forehand & Frame, 1992). 
It has been postulated by several theorists that delinquent 
behavior is a higher order form of conduct disorder, with t.\e 
offenses being more severe (Loeber, 1990; Patterson, DeBaryske, & 
Ramsey, 1989; Thornberry, 1987). 

Substance Abuse . A number of recent studies found a 
significant relationship between emotional/behavioral disorders 
and substance abuse in psychiatric adolescent samples (Bukstein, 
Glancy & Kaminer, 1992; Kutcher, Marton, & Korenblum, 1989). 
Neighbors, Kempton, and Forehand (1992) found that the diagnosis 
of abuse or dependence on alcohol or drugs was associated with a 
diagnosis of conduct disorder in over 90% of the sample; a co- 
diagnosis of depression and anxiety occurred in 38% and 30% of 
the sample, respectively. When polysubstance abuse was present 
(alcohol and drugs) , the probability of having more than one of 
the three diagnoses (conduct disorder, depression, or anxiety) 
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was above 50%. 

In relation to prograiTi outcomes, alcohol and substance abuse 
were found to be associated with program success and failure for 
adolescents with E/BD in a vocational training and placement 
program (Bullis et al., 1994). 

Gender and Ethnicity. These two variables were necessarily 
included, since established research has demonstrated various 
degrees of association with outcomes for students with 
disabilities. Wagner (1992), in a synthesis of findings from the 
National Longitudinal Transition Study, concluded that females 
with disabilities in secondary special education tended to be 
more seriously impaired than males with disabilities. 

The variable of ethnicity has been one of the most 
controversial issues in special education. Overrepresentation of 
minority students in special education classes has been 
documented (Finn, 1982) and litigated ( Larry P. v. Riles , 1979). 

Method 

Sample 

The subjects in this study were 134 students identified as 
E/BD and received services in a secondary level, therapeutic day 
treatment program in a state approved, nonpublic, special 
education facility. A description of this facility follows. 
Since 1988, student records at this facility have been entered 
into a database which included demographic, educational, 
psychological, and parental/social characteristics of all 
students. 
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According to archival records and database information, of 
the 258 students who attended this facility, a total of 134 
students have been discharged to date. Of those, 47 were 
discharged successfully and 87 unsuccessfully. Successful 
discharges were considered to include graduation, certificate of 
completion, or transition into a regular classroom setting. 
Unsuccessful discharges included assault/aggressiveness, drug 
related dismissal, failed senior year, long term hospitalization, 
insufficient progress/ lack of effort, pregnancy related 
withdrawal, refusal to attend/poor attendance/dropped out, 
withdrawn by parent, withdrawn by self, and more restrictive 
environment required. Students who moved were not included in 
this sample. 

The discharged student population was 78% male and 22% 
female; the ethnic background of students was 35% minority 
(Black, Hispanic, and Indian) and 65% Caucasian. The average 
number of prior school placements, before coming to the 
educational facility, was 3.84 (SD = 2.19). The average age at 
admission was 16.14 years (SD = 2.65) and the average grade at 
admission 7.69 (SD = 1.89). The average grade at discharge was 
8.67 (SD = 2.16) and the average time enrolled in the educational 
facility was 1.81 years (SD = 1.58). 
Setting 

The setting was a therapeutic educational treatment facility 
serving adolescents with E/BD. All of the students served at the 
facility were referred and funded by local public schools. 

o 1 a 
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AERA: epidemiology evaluation 13 
students enrolled in this facility met each of the following 
criteria: (a) had a primary diagnosis of severe emotional 
disturbance; (b) were in at least 6th grade upon referral 
(program includes grades 6 - 12); (c) had verbal IQs greater than 
or equal to 80 or achievement scores no more than two years 
behind expected grade level in reading and math; (d) were not in 
need of physical containment, nor homicidal, nor a threat to the 
community, and (e) had no previous conviction for distribution or 
possession with intent to distribute a controlled substance. 
Services provided in this setting included: (a) peer mediation; 
(b) state student assistant program; (c) behavior management 
system; (d) individual therapy once a week; (e) family therapy; 
(f) drug and substance abuse counseling; (g) art therapy; (h) 
music therapy; and (i) psychodrama therapy. 
Outcome Variable 

Since the purpose of the study was to identify and quantify 
risk factors related to unsuccessful outcome for students with 
E/BD, the outcome variable selected for examination was 
unsuccessful program discharge. For purposes of this paper, 
successful discharge (SD) was defined as a 'good' outcome and 
unsuccessful discharge (UD) as a 'poor' outcome. The analyses 
described below investigated which variables were associated with 
UD, or 'poor' outcome. 
Predictor Variables 

Five variables were selected as predictor variables for 
investigation, based on their potential relationship to student 



AERA: EPIDEMIOLOGY EVALUATION 14 
discharge status and their documentation in school records. 
Selected variables included Juvenile Services involvement, 
truancy, gender, and ethnicity. 

This type of epidemiologic analysis required that predictor 
variables be dichotomous, modifications of some variables were 
required. The variable of ethnicity was restricted to Black and 
Caucasian students only. Six students of other ethnic 
backgrounds were not included in the data set, which reduced the 
total number of subjects included in the study to 128. 

Students were identified as 'yes' for truancy if they had 
been truant (more than 9% unexcused absences) prior to admission 
to this facility or while enrolled in this facility. A rating of 
'yes' for the variable of student substance abuse was documented 
through a student's educational records, a counselor's report, or 
reference to hospitalization for substance abuse. Juvenile 
services involvement was documented through educational records, 
although the exact nature of the involvement was not available. 
Analyses 

Two sets of data analyses were conducted. The first set, 
univariate analyses, provided the risk of unsuccessful discharge 
(UD) for each of the predictor variables. The second set of data 
analyses included the calculated values of the additive effects 
of pairs of predictor variables on the outcome variable of UD. 
Both the univariate and additive analyses required the 
computation and interpretation of two measures, the Exposure 
Frequency and the Odds Ratio. 
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Exposure Frequency , What proportion of subjects in the 
total sample were identified as having the predictor variable? 
In order to examine the importance or weight of the relative risk 
estimates provided by epidemiologic analysis, one needs to 
consider the value of risk relative to the proportion of subjects 
in the totsl sample who possessed the variable. Thus the 
calculation of the frequency of the occurrence of the variable, 
or the Exposure Frequency (EF) , may be used as a descriptive and 
interpretive tool, reflecting the occurrence of the predictor 
variable in the sample. If few subjects had the predictor 
variable present the resulting low EF may indicate a low 
prevalence or relatively rare variable, or may indicate a 
sampling error. Likewise, a high occurrence may also indicate 
high prevalence or a sampling error. The EF informs the 
researcher of the rate at which the variable occurs in the sample 
and aids in interpretation of the Odds Ratio. 

Odds Ratio . Of the subjects determined to have experienced 
a poor outcome, what proportion were identified as having the 
predictor variable? The calculation of the Odds Ratio (OR) is 
made to determine the likelihood, or odds, that a discrete 
outcome (UD v. SD) is associated with a specific level of a 
predictor variable (present v. absent) . The OR is the ratio of 
the occurrence of the discrete outcome of interest when the level 
of the predictor variable is present compared to the discrete 
outcome when the level of the predictor variable is absent. An 
interpretation may then be made regarding the elevated risk of 
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the discrete outcome when the level of the risk variable is 
present. 

The analytic concept of risk is quantified by a numerical 
value representing degree of at-risk status. An OR greater than 
1.00 would indicate a positive association, or an increased risk 
of poor outcome among those exposed to the predictor variable. 
Conversely, if the OR were less than 1.00, an inverse 
relationship would be determined to exist between the predictor 
variable and outcome. Thus, an OR calculated to equal 1.4, would 
be interpreted to mean that the risk of unsuccessful discharge is 
40 percent (i.e., 1.4 minus the null value of 1.0) higher in 
students identified as 'yes' for substance abuse compared to 
students who have been identified as 'no' for substance abuse. 
Further, an OR equal to 2.0 indicates that subjects with the risk 
variable present are 100 percent (i.e., 2.0 minus 1.0), or 2 
times more likely to experience a poor outcome compared to 
subjects without the risk variable (Henneckens & Burning, 1987) . 

The OR calculation is a statement of probability of outcome, 
with magnitude implying strength of association and degree of 
risk status. Therefore, an OR states the probability of an 
outcome in one group (risk variable present) as compared to a 
second group (risk variable absent) . For example, if an OR is 
equal to 2 . 0 for unsuccessful outcome in students identified as 
'yes' for substance abuse, this indicates that students 
identified as 'yes' for substance abuse are 2 times more likely 
to have experienced unsuccessful discharge in comparison to 
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students identified as 'no' for substance abuse. This value is 
not interpreted using a criterion of level of significance (i.e., 
E < .05). Instead, any value greater than 1.0 represents an 
elevated risk status. For this study, all ORs will be reported; 
however, since there were so many variables with computed ORs 
greater than 1.00, only ORs greater than 2.0 will be presented as 
an elevated risk status. 

As discussed above, it is important to interpret the OR in 
consideration of the EF level. If the EF is small, only a small 
number of the sample have experienced the variable. A low EF 
with a correspondingly low OR should not necessarily be 
interpreted to indicate a poor predictor variable; rather, the 
low EF T^.ay indicate the need for a larger sar.ple. In contrast, 
however, if the EF is small and the estimated risk OR is large, 
this indicates that the variable has a dramatic impact when 
present, even in a small portion of the sample. 

Results 

Univariate Analyses 

Two types of univariate analyses are presented in Table 2, 
descriptive and epidemiologic. Descriptive chi-square analyses 
were performed on each predictor variable testing relationship s 
with the outcome variable of discharge. This was done to provide 
the reader with a familiar statistical test for a criterion 
reference. Significant differences were found for the variables 
Juvenile Services and Student Substance Abuse with the presence 
of affirmative involvement for either variable related to UD (see 
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Table 2 for chi~square values) . Nearly 80 percent of students 
who were involved with Juvenile Services had UD outcomes and 85 
percent of students identified or suspected of substance abuse 
were unsuccessfully discharged. 

The epidemiological analysis results were similar to the 
chi-square findings. Juvenile Services and student substance 
abuse both had a low EF of near 20 percent in this sample, yet 
both had elevated Ors at 2.79 and 3.70, respectively. This 
indicates that even though the exposure of Juvenile Services and 
substance abuse involved less than a quarter of the entire 
sample, involvement with either one was associated with an 
elevated risk of UD from this program. In other words, the 
presence of any one of these predictor variables placed students 
at nearly a three times higher risk for not successfully 
completing the program compared to students who did not have 
these characteristics. 
Additive Effects of Risk Factors 

In an additive model, the combined effects of two risk 
factors are analyzed. One variable is selected for control and 
subjects with that variable present are subsampled from the 
population. Using the selected subsample, the other risk factor 
variables are each then put into the format of Table 1, the 2x2 
criterion table, with the EF and OR calculated. In this type of 
analysis, specific indicators of one risk factor are controlled 
by selecting only subjects from the sample who have that 
characteristic. Then the additive effect of a second risk factor 
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may be determined by examining the occurrence of levels of the 
second risk factor in the select sample and calculating an EF and 
OR. Despite the individual effects of a risk factor as 
estimated in the univarie'^.e analysis, the additive effect of a 
second risk factor to a specific level of the first variable may 
reveal surprising effects as two characteristics are combined. 
The results of the additive effects of combined risk factors are 
found in Table 3 and will be discussed in order of gender, 
ethnicity, Juvenile Services, truancy, and substance abuse. 

Gender . The first section of Table 3 examined the additive 
effects of the other predictor variables for female subjects. 
Results of this analysis found Caucasian female subjects were 
nearly 7 times more likely to have an UD than Black female 
subjects. The EF indicates that ethnicity was not skewed, since 
there were 55% Caucasian and 45% Black females. Similarly, 
females who were also truant were nearly 7 times more likely to 
have UD (EF = 55%) , compared to females who were not truant. 

Other additive variables for female subjects which increased 
the risk of UD were substance abuse (OR == 5.5; EF == 21%) and 
Juvenile Services involvement (OR == 3.38; EF = 14%). Of special 
significance is the fact that, when female subjects were involved 
with either of these variables, the outcome was always UD. While 
the EF of both Substance Abuse and Juvenile Services were low, 
21% and 14% respectively, both were demonstrated to be definitive 
of unsuccessful discharge for female students. 

When male students were examined, the additive effect of 
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other predictor variables were not as negative as for females. 
Elevated risk was found only for male subjects who were involved 
with substance abuse (OR = 2.50; EF = 20%) or Juvenile Services 
(OR = 2.07; EF = 26%). Both of these variables had relatively 
low Efs, therefore, these variables should be considered as 
strong risk indicators for UD. Males who came from two non-birth 
parent families were also at an elevated risk (OR = 2.69), but 
the EF was very large (EF = 75%) , urging caution in 
interpretation . 

Ethnicity . When the discharge status of Black students was 
examined in the additive model, the only elevated risk was found 
for Black male students. These students were 3 times more likely 
(OR = 3.05; EF = 71%) to have an unsuccessful discharge than 
Black female students. As shown in Table 3, there was little 
identifiable risk associated with the other predictor variables 
for Af rican-Am.erican students. 

Caucasian students, however, were found to be at 
substantially greater risk for UD when additive risk factors were 
computed. As identified through the gender analysis described 
above, Caucasian female students were at an elevated risk for UD 
(OR = 2.32; EF = 20%) compared to Caucasian male students. 
Regardless of gender, Caucasian students who had been involved 
with substance abuse (OR = 5.54) or Juvenile Services (OR = 4.06) 
were clearly at high risk, as evidenced by the high OR relative 
to the low EF. This indicates that while only approximately 25% 
of the Caucasian subjects had these characteristics, these 



AERA: EPIDEMIOLOGY EVALUATION 21 
Students had significantly poorer outcomes than students without 
the variable. The final elevated risk factor for Caucasian 
students was family living status (OR = 3.08; EF = 75%)* 

Juvenile Services . Looking at the additive effect of the 
other risk variables for students who were involved with Juvenile 
Services indicated that Caucasian students who had Juvenile 
Services involvement were at a higher risk for UD (OR = 4.50; EF 
== 75%) than Black students with Juvenile Services involvement. 
In addition, students who had been involved with Juvenile 
Services and abused were at an elevated risk (OR =2.00; EF = 
46%) compared to those involved with Juvenile Services but not 
abused. 

Truan t . Students who had been identified as being truant 
either before or following admission had an elevated risk of UD 
if they were involved with substance abuse (OR = 3.25) or if they 
were female (OR = 3.01) . Both of these additive variable 
combinations evidt iced relatively low Efs, 30% and 29% 
respectively, which highlights the importance of these 
characteristics relative to program outcome. 

Substance Abuse . Students identified or suspected of 
substance abuse were at greatest risk for UD if they were 
Caucasian (OR = 4.50; EF = 77%). In the univariate analysis, 
substance abuse was significantly related to poor outcome, with 
85% of students with this characteristic having UD. In the 
additive models presented earlier in this section, this variable 
was also shown to increase the risk of poor outcome for gender, 

O r 

ERLC 
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ethnicity, and truancy. 

Summary . The additive effects of the combined risk factors 
of gender, ethnicity. Juvenile Services involvement, truancy, and 
substance abuse yielded several variable pairs which produced 
extremely elevated levels of risk related to poor outcome, UD, in 
this program. Table 4 presents a summary of selected variable 
pairs with three to seven times greater risk for UD, prioritized 
in order of greatest risk elevation. These and other selected 
findings will be discussed. 

Discussion 

The results of this study identified several risk factors 
associated with poor outcome in an educational program for 
children and adolescents with behavior disorders. Univariate 
risk factors related to unsuccessful completion of the program 
were Juvenile Services involvement and student substance abuse. 
The additive relationships of the risk variables was more 
informative. 

Gender differences were found for additive risk factors of 
UD. Females who were also truant were at elevated risk of UD 
compared to females who were not truant; truancy did not have an 
additive effect upon risk status for males, which was less than 
1.00. 

Ethnic differences related to gender were very informative. 
Caucasian students were at elevated risk for UD compared to Black 
students (OR = 1.60); however, when the additive effect of gender 
was controlled, only Caucasian Females were shown to be a high 
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risk group (OR = 6.93). As the EF indicates, the 'frequencies of 
Black and Caucasian females were similar and indicated no skew. 
Interpretation is, however, speculative for this sample. 

Ethnic differences were found for the additive variables of 
Juvenile Services and substance abuse. Caucasian students with 
either of these characteristics were at 4 and 5 times the risk of 
UD; Black students with either of these characteristics wore 
found to be at slight risk for UD. Efs were lower in the Black 
sample for these risks, indicating limited involvement. 

Results of this investigation are applicable to the outcome 
selected for the specific facility under investigation. In this 
case, it has been demonstrated that students with E/BD served in 
one therapeutic treatment facility who have selected 
characteristics have an elevated risk of poor outcome compared to 
students without selected characteristics. The results of the 
epidemiological analysis is situational specific to this program, 
and caution regarding external validity is urged. 

In the program evaluated in this study, the data suggested 
that female truants needed greater attention from the program 
planners and administrators. Response to this analysis is 
currently being implemented in the setting where the data was 
collected. 

When female students have an unexcused absence, a phone call 
is made to the student's home to inquire about their absence. A 
phone call is also made to the student's parents. If the pattern 
of unexcused absences continues, the school arranges for a parent 
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conference to discuss the student's truancy problems. This is a 
daily routine. 

The school staff have been vigilant about students' 
attendance in school. This attempt to reduce truancy will 
hopefully swing the OR of outcome toward successful discharge. 
This determination will be made with annual evaluation of the 
outcome status and relation to the risk variables. 

In summary, epidemiologic analysis affords practitioners and 
program developers the opportunity to investigate the correlation 
of risk characteristics and program effectiveness. In this 
study, risk factors have been identified and quantified. The 
outcome variable has been specified and quantified. Findings 
provide specific information on subpopulations which require 
modifications or alternative interventions. After programmatic 
interventions have been devised and implemented, new odds ratios 
may be compared to the prior odds ratios, looking for change. If 
the new odds ratios calculations and exposure frequencies are 
smaller than the prior ones, this may indicate a decrease in the 
number of UD (poor outcome) students, indicating support for the 
program change implemented. To confirm the validity of the 
findings, however, data from sequential years should be analyzed 
and programmatic changes documented. 

Other programs may have similar or dissimilar findings based 
on specific program elements. As described above, this seeming 
limitation underscores the usefulness of epidemiologic analysis 
as a simple program evaluation tool which provides customized, 



AERA: EPIDEMIOLOGY EVALUATION 25 
readily useful information to practitioners. Despite concurrence 
of this study's with other studies' findings, additional 
evaluations of programs serving similar populations are 
recommended to assess the generalizability of these findings and 
to begin to identify specific program elements related to 
successful or unsuccessful outcomes for students. Further, 
epidemiologic analysis may enable cross-program comparisons 
related to specific outcomes for subpopulations and may provide 
data to assist with placement decisions for the very 
heterogeneous group of students identified as seriously 
emotionally disturbed. 
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Table 1 

Predictor-Criterion 2x2 Table. 



Predictor 


Criterion 


(outcome) 




Variable 


Poor 
(UD) 


Good Total 
(SD) 


Level 1 
(poor) 


A 

True Positive 


B 

False Positive 


Ml 


Level 2 
(good) 


C 

False Negative 


D 

True Negative 




Total 


Ni 


N2 


Nt 



Note, UD = Unsuccessful Discharge; SD = Successful Discharge 



Epidemiologic Formulas 

Exposure Frequency (EF) = Mi / N^ 

Odds Ratio (OR) = (A / C) / (B / D) = AD / BC 



oi 



Table 2 



Frequency Counts of Outcome Variable and Epidemiologic 
Calculations for Predictor Variables. 



Discharge Status Epidemiologic 

Predictor UD SD Analysis 

Variables 

N (%) N (%) EF OR 



Total Subjects 83 (65) 45 (35) 
Gender 

Female 18 (62) 11 (38) .22 0.86 

Male 64 (65) 34 (35) .78 1.15 

Ethnicity 

White 57 (69) 26 (31) .65 1.60 

African-American 26 (58) 19 (42) .35 0.62 

Juvenile Services Administration Involvement' 

Yes 22 (79) 6 (21) .24 2.79 

None 50 (57) 38 (43) 

Student Substance Abuse^ 

Yes 22 (85) 4 (15) .20 3.70 

None 61 (60) 41 (40) 

Truancy 

Yes 37 (66) 19 (34) .45 1.11 

No 44 (64) 25 (36) 

V(1,N=116)=4.27, E < .05 

V(1,N=128)=5.59, E < .05 



Note. UD = Unsuccessful Discharge; SD = Successful Discharge 
= Exposure Frequency; OR = Odds Ratio. 



Table 3 

The Additive Relation of Paired Risk Variables to Unsuccessful 



Discharge. 









Discharge 


Status 


Epidemiologic 








UD 


SD 


Analysis 


Variables 


N 


N 


EF 


OR 


Female 


& Caucasian 


13 


3 


.55 


6.93 






Black 


5 


8 


.45 


0.14 


Female 


& JS (yes) 


4 


0 


. 14 


3 . 38 






JS (no) 


13 


11 






Female 


& Truant (yes) 


13 


3 


. 55 


6.93 






Truant (no) 


5 


8 






Female 


& SubAbuse (yes) 


6 


0 


.21 


5.50 






SubAbuse (no) 


12 


11 






Male 


& 


Caucasian 


43 


23 


.67 


0.98 






Black 


21 


11 


.33 


1.02 


Male 


& 


JS (yes) 


17 


6 


.26 


2.07 






JS (no) 


37 


27 






Male 


& 


Truant (yes) 


23 


16 


.41 


0.63 






Truant (no) 


39 


17 






Male 


& 


SubAbuse (yes) 


16 


4 


.20 


2 .50 






SubAbuse (no) 


48 


30 







Note. UD = Unsuccessful Discharge; SD = Successful Discharge; 
EF = Exposure Frequency; OR = Odds Ratio, JS = Juvenile Services 
involvement. 
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Table 3^ cont. 





Discharge 


Status 


Epidemiologic 






TTn 


SD 


Analysis 


Variables 




N 


N 


EF 


OR 


Black fit 


Female 


5 


8 


.29 


0.33 




wa j.e 




11 


.71 


3.05 


Black St 


JS (yes) 


4 


3 


. 18 


1.25 




JS (no) 




15 






Black St 


Truant (yes) 


8 


5 


.30 


1.31 




Truant (no) 


1 7 


14 ■ 






Black St 


SubAbuse (yes) 


4 


2 


.13 


1.54 




SubAbuse (no) 




17 






Caucasian St 


Female 


13 


3 


.20 


2.32 


Male 


43 


23 


.80 


0 . 43 


Caucasian St 


JS (yes) 


18 


3 


.27 


4.06 


JS (no) 


34 


23 






Caucasian St 


Truant (yes) 


29 


14 


.53 


0. 84 


Truant (no) 


27 


11 






Caucasian St 


SubAbuse (yes) 


18 


*-> 

2 


.24 


5. 54 


SubAbuse (no) 


39 








JS St Female 


4 


0 


. 15 


1.41 


Male 




17 


6 


.85 


0. 71 


JS St Black 




4 


3 


.25 


0.22 


Caucasian 


18 


3 


.75 


4.50 


JS St Truant (yes) 


9 


5 


.52 


0. 15 


Truant (no) 


12 


1 







JS & SubAbuse (yes) 5 2 .25 0.59 

SubAbuse (no) 17 4 



Table 3, cont. 



Discharge Status Epidemiologic 



Variables 


UD 


SD 


Analysis 


N 


N 


EF 


OR 


Truant 


& Female 


13 


3 


.29 


3.01 




Male 


23 


16 


.71 


0.33 


Truant 


& Black 


8 


5 


.23 


0.77 




Caucasian 


29 


14 


.77 


1.29 


Truant 


& JS (yes) 


9 


5 


.26 


1.01 




JS (no) 


25 


14 






Truant 


& SubAbuse (yes) 


14 


3 


.30 


3.25 




SubAbuse (no) 


23 


16 







SubAbuse 


& 


Female 


6 


0 


.23 


1.50 






Male 


16 


4 


.77 


0.67 


SubAbuse 


& 


Black 


4 


2 


.23 


0.22 






Caucasian 


18 


2 


.77 


4.50 


SubAbuse 


& 


JS (yes) 


5 


2 


. 35 


0.21 






JS (no) 


12 


1 






SubAbuse 


& 


Truant (yes) 


14 


3 


.65 


0.58 






Truant (no) 


8 


1 







GO 



Table 4 

Summary of Paired Risk Variables with Odd Ratios greater than 3.0. 



Variable Pairs OR EF 



Female and Caucasian 


6.93 


.55 


Female and Truant 


6.93 


.55 


Caucasian and Substance Abuse 


5.54 


.24 


Female and Substance Abuse 


5.50 


.21 


Juvenile Services Involved and Caucasian 


4.50 


.77 


Female and Juvenile Services Involved 


3.38 


.13 


Truant and Substance Abuse 


3.25 


.31 


Black and Male 


3.05 


.71 
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